{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3XQZGcZNb2jR"
      },
      "source": [
        "## PolyBERT: Introduction to Chemical Language Models for Fingerprint Generation\n",
        "By Debasish Mohanty"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NYxIpu7tcF_0"
      },
      "source": [
        "### Table of Content\n",
        "1. [Introduction](#intro)\n",
        "2. [Overview of PSMILES and Fingerprints](#overview)\n",
        "3. [Understanding PolyBERT Latent Space](#polybertlspace)\n",
        "4. [Implementing PolyBERT using HuggingFace](#hfimplement)\n",
        "5. [Similarity Predictions Using Fingerprint](#similarpred)\n",
        "6. [Summary](#sum)\n",
        "7. [Reference](#ref)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TwXifdUwcSvW"
      },
      "source": [
        "### 1. Introduction <a id=\"intro\"></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYMg81QkeeqB"
      },
      "source": [
        "This article introduces PolyBERT, an innovative machine learning approach to polymer informatics inspired by natural language processing techniques. PolyBERT enables researchers to efficiently and accurately search for optimal polymer candidates within this vast chemical space. By employing fingerprints PolyBERT translates polymer chemical structures into a machine-interpretable vector.\n",
        "The PolyBERT can have various applications from chem-informatics to multitask property predictions. It can be employed in different pipelines to generate or predict relevant polymer molecules and properties as shown in Figure -1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yD8w6nqusd50"
      },
      "source": [
        "![polybert_intro](https://media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41467-023-39868-6/MediaObjects/41467_2023_39868_Fig1_HTML.png?as=webp)\n",
        "\n",
        "**Figure - 1 :** The pipelines and applications of PolyBERT `[1]`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S-U6kF5SjeFV"
      },
      "source": [
        "### 2. Overview of PSMILES and Fingerprints <a id=\"overview\"></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJAKEpsXsne-"
      },
      "source": [
        "Polymer SMILES (PSMILES) `[1]` is an extension of the Simplified Molecular Input Line Entry System (SMILES) notation, specifically designed to represent polymer structures. While traditional SMILES is used for small organic molecules, PSMILES accounts for the repeating units characteristic of polymers. This notation uses the wildcard `[*]` to indicate the points where the polymer chain extends, enabling the representation of large macromolecules in a concise and machine-readable format.\n",
        "\n",
        "Key Features of PSMILES:\n",
        "- **Repeating Units**: PSMILES captures the repeating units in a polymer, essential for understanding polymer behavior and properties.\n",
        "- **Wildcard Symbol `[*]`**: Marks the connection points of the polymer's repeating unit.\n",
        "- **Computational Use**: Facilitates the analysis, storage, and sharing of polymer data in cheminformatics and materials science.\n",
        "\n",
        "PSMILES provides a standardized way to describe polymers, making it easier to perform computational studies and design new materials with specific properties."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hppYYzpQx48q"
      },
      "source": [
        "Chemical fingerprints are binary or integer vectors that represent the presence or absence of specific substructures or patterns within a molecule. These fingerprints serve as molecular descriptors, summarizing a molecule’s structure in a format that is suitable for computational tasks such as similarity searches, clustering, and machine learning applications.\n",
        "\n",
        "Key Features of Chemical Fingerprints:\n",
        "\n",
        "- **Similarity Search:** Identifying compounds with similar structures.\n",
        "- **Clustering and Classification:** Grouping molecules based on structural similarities.\n",
        "- **Machine Learning:** Used as input features in predictive models for drug discovery and materials science.\n",
        "\n",
        "Chemical fingerprints are powerful tools for transforming molecular structures into data that can be efficiently processed and analyzed, making them essential in modern cheminformatics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MZrP72ZO0jG0"
      },
      "source": [
        "PSMILES are beneficial for streamlined data processing and sequential machine learning models, whereas fingerprints provide a wide range of scientific utilities, from searching for similar molecules to facilitating material discovery. In this tutorial, we have demonstrated how to generate equivalent fingerprints (using transformer latent space) from PSMILES string representations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFh0q9ZijdsO"
      },
      "source": [
        "### 3. PolyBERT Latent Space <a id=\"polybertlspace\"></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n_bKkjyQ3ZQo"
      },
      "source": [
        "Latent space refers to a lower-dimensional representation of data, where complex information is encoded in a more compact form. In the context of transformers, latent space is where the model captures essential features and patterns of input data, like PSMILES strings, after processing through several layers. This space allows the model to focus on important relationships and characteristics that are useful for tasks like predicting properties, clustering, or generating new data.\n",
        "\n",
        "[PolyBERT](https://www.nature.com/articles/s41467-023-39868-6) `[1]` is a chemical language model developed upon DeBERTa architecture. [DeBERTa](https://arxiv.org/abs/2006.03654) (Decoding-enhanced BERT with Disentangled Attention)`[2]` is an advanced transformer architecture that improves how information is processed. Unlike standard transformers, DeBERTa disentangles the position of words (or atoms in a molecule) and their content, leading to more accurate representations. When PolyBERT uses the DeBERTa architecture, it leverages these improvements to better understand and generate the latent space for polymers, making it more effective in tasks like generating chemical fingerprints or predicting molecular properties.\n",
        "\n",
        "PolyBERT leverages the latent space of the DeBERTa architecture to attain the functionalities of fingerprints. PolyBERT specifically uses fingerprint of **600** dimensions making the prediction and clustering tasks highly accurate. Once implemented, the fingerprints can be mapped with properties of the polymers to do multi-task predictions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oET2xow1j9Z6"
      },
      "source": [
        "### 4. Implementing PolyBERT using HuggingFace <a id=\"hfimplement\"></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hqOBuXXfPYxy"
      },
      "source": [
        "The tokenizer and the model weights of the PolyBERT can be found in [HuggingFace](https://huggingface.co/kuelumbus/polyBERT) model repository. For our case we will take two PSMILES i.e. PolyEthylene (PE) and PolyVinylAlcohol(PVA) represented as `[*]CC[*]`, `[*]COC[*]` respectively. We will use PolyBERT to generate fingerprints for both PSMILES using following steps.\n",
        "\n",
        "1. Load the tokenizer and model weights for PolyBERT\n",
        "2. Tokenize the PSMILES and generate the fingerprint\n",
        "3. Performing the mean-pooling (to normalize the fingerprint value)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CIbv7yHqSxvg"
      },
      "source": [
        "#### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "LRgWjM1WS1_4",
        "outputId": "9660f8d4-613d-4adb-d1a8-474e798ebf56"
      },
      "outputs": [],
      "source": [
        "! pip install rdkit-pypi transformers\n",
        "from transformers import AutoTokenizer, AutoModel\n",
        "import torch\n",
        "from rdkit import Chem\n",
        "from rdkit.Chem import Draw\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86uQPg_QWOkY"
      },
      "source": [
        "#### Visualizing Polymers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "id": "j8-FrAsnT2hR",
        "outputId": "63c45ae2-858e-44d9-f785-70e010e26dcd"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAADaCAYAAABq1w8LAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAApYklEQVR4nO3deVxU5f4H8M+ACAq4oGjqVRAQSMX9quGCa4hLLJpKimJqmlfNtbRQ3LrmLdcsy/CXhmCAuOCCphfMXNIs9yXRcE1FBZREYJjz/f0xzVwRUdQjA/h5v16+ipkzZ55z5jzzOc9yzmhEREBERKQiM1MXgIiISh+GCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hguADh06oEOHDiZ57+DgYNjY2Ki2vgsXLkCj0WDlypWqrZNeXi+6bqxcuRIajQYXLlwolu8RHBwMR0dH1ctk4OjoiJ49e6q2vuJU/0tsuBgOGMM/KysruLq6YvTo0bhx44api5dHZmYmZsyYgV27dpm6KPQSKEl1g0qvMqYuwPOaNWsW6tati6ysLOzZswfLli3D1q1bceLECZQvX97UxQOgD5eZM2cCgMlaSPTyKQl1IygoCP3794elpaWpi0IqK/Hh4uPjgxYtWgAAhg0bhipVqmDBggXYuHEjAgMDTVw6ItMpCXXD3Nwc5ubmpi4GvQAltlusIJ06dQIAJCcnIzc3F7Nnz4azszMsLS3h6OiIDz/8ENnZ2QW+/q+//oK1tTXee++9fM9duXIF5ubmmDt3rvGx9PR0jBs3DrVr14alpSVcXFwwb948KIoCQN8Ham9vDwCYOXOmsatixowZedZ99epV+Pn5wcbGBvb29pg0aRJ0Oh0AQETg6OgIX1/ffGXKyspCxYoVMWLEiMfulzNnzqBPnz6ws7ODlZUVWrRogbi4uDzLGLpT9u7diwkTJsDe3h7W1tbw9/fHzZs3860zPj4e7dq1g7W1NWxtbdGjRw+cPHnyseUg0ymqunHo0CFoNBqsWrUq33Lbt2+HRqPB5s2bATx6PMQwDrFnzx60bNkSVlZWcHJywnfffWdc5o8//oBGo8HChQvzvce+ffug0WiwZs2aArdl48aN6NGjB2rWrAlLS0s4Oztj9uzZxjr3OIqiYPHixfDw8ICVlRXs7e3RrVs3HDp0yLjM0+7fx23rg9v85ptvws7ODuXLl0fr1q2xZcuWJ5bXVEpduJw/fx4AUKVKFQwbNgzTp09Hs2bNsHDhQnh5eWHu3Lno379/ga+3sbGBv78/oqKi8h1oa9asgYhgwIABAPTdXV5eXli9ejUGDRqEJUuWoE2bNpg6dSomTJgAALC3t8eyZcsAAP7+/ggPD0d4eDgCAgKM69XpdPD29kaVKlXw2WefwcvLC/Pnz8fy5csBABqNBgMHDkR8fDxSU1PzlGnTpk24e/cuBg4cWOA2nTx5Eq1bt8bp06cxZcoUzJ8/H9bW1vDz88P69evzLT9mzBgcPXoUoaGhePfdd7Fp0yaMHj06zzLh4eHo0aMHbGxsMG/ePEybNg2nTp1C27ZtX+jgLD27oqobLVq0gJOTE6Kjo/OtIyoqCpUrV4a3t/djy3ru3Dn06dMHXbt2xfz581G5cmUEBwcbT16cnJzQpk0bRERE5HttREQEbG1tH3kyZrBy5UrY2NhgwoQJWLx4MZo3b47p06djypQpjy0XAAwdOtR4Qjlv3jxMmTIFVlZW+Pnnn43LPM3+fdK2AsCNGzfg6emJ7du3Y9SoUfj444+RlZWFN95445F1uFiQEurbb78VALJz5065efOmXL58Wb7//nupUqWKlCtXTnbt2iUAZNiwYXleN2nSJAEgCQkJxse8vLzEy8vL+Pf27dsFgMTHx+d5baNGjfIsN3v2bLG2tpazZ8/mWW7KlClibm4uly5dEhGRmzdvCgAJDQ3Ntx2DBw8WADJr1qw8jzdt2lSaN29u/Pv3338XALJs2bI8y73xxhvi6OgoiqKIiEhycrIAkG+//da4TOfOncXDw0OysrKMjymKIp6enlKvXj3jY4Z92qVLF+P6RETGjx8v5ubmkp6eLiIiGRkZUqlSJRk+fHiesly/fl0qVqyY73EqWsWhbkydOlUsLCwkNTXV+Fh2drZUqlRJ3n777XxlTU5ONj7m4OAgAGT37t3Gx1JSUsTS0lImTpxofOzrr78WAHL69GnjYzk5OVK1alUZPHjwY98jMzMz334bMWKElC9fPk89GTx4sDg4OBj/TkhIEAAyduzYfK831JkjR44Uev8WdlvHjRsnAOSnn34yPpaRkSF169YVR0dH0el0IvLo+m8qJb7l0qVLF9jb26N27dro378/bGxssH79euzbtw8AjC0Ig4kTJwLAY5uTXbp0Qc2aNfOcFZ04cQLHjh3L00KIiYlBu3btULlyZdy6dcv4r0uXLtDpdNi9e3eht2PkyJF5/m7Xrh3++OMP49+urq5o1apVnjKlpqYiPj4eAwYMgEajeeR6U1NTkZCQgL59+yIjI8NYxtu3b8Pb2xtJSUm4evVqnte88847edbXrl076HQ6XLx4EQCwY8cOpKenIzAwMM92m5ubo1WrVkhMTCz0dtOLY8q60a9fP2i1Wqxbt8742A8//ID09HT069fviWWvX78+2rVrZ/zb3t4ebm5ueepE3759YWVllacs27dvx61btx7bkgeAcuXKGf/fUC/atWuHzMxMnDlzpsDXxcbGQqPRIDQ0NN9zhjqzdetWAIXfv4XZ1q1bt6Jly5Zo27at8TEbGxu88847uHDhAk6dOvXY7TWFEj+g/8UXX8DV1RVlypRB9erV4ebmBjMzM6xfvx5mZmZwcXHJs/wrr7yCSpUqGb8oH8XMzAwDBgzAsmXLkJmZifLlyyMiIgJWVlZ48803jcslJSXh2LFjxjGVh6WkpBRqGwz9tg+qXLky0tLS8jw2aNAgjB49GhcvXoSDgwNiYmKg1WoRFBRU4LrPnTsHEcG0adMwbdq0AstZq1Yt49916tTJVxYAxvIkJSUB+F8f/sMqVKhQYHmo6JiybjRu3Bju7u6IiorC0KFDAei7xKpWrVrgcfOgh49BIH+dqFSpEnr16oXIyEjMnj0bgL5LrFatWk98j5MnTyIkJAQJCQm4e/dunufu3LlT4OvOnz+PmjVrws7OrsBlLl68+FT7tzDbevHiRbRq1Srfcq+++qrx+YYNGxZYJlMo8eHSsmVL44yYRynojP5JBg0ahE8//RQbNmxAYGAgIiMj0bNnT1SsWNG4jKIo6Nq1K95///1HrsPV1bVQ71XY2TL9+/fH+PHjERERgQ8//BCrV69GixYt4ObmVuBrDBMLJk2aVGA/98OVoKDyyN+/iG1YZ3h4OF555ZV8y5UpU+IPq1LBlHUD0LdePv74Y9y6dQu2traIi4tDYGBgoY6PJx2DD5YlJiYG+/btg4eHB+Li4jBq1CiYmRXcKZOeng4vLy9UqFABs2bNgrOzM6ysrPDbb7/hgw8+MB7fz6uw+7ew21rSlNpvAQcHByiKgqSkJGO6A/qBsfT0dDg4ODz29Q0bNkTTpk0RERGBf/zjH7h06RI+//zzPMs4Ozvjr7/+QpcuXR67rmetxA+zs7NDjx49EBERgQEDBmDv3r1YtGjRY1/j5OQEALCwsHhiOQvL2dkZAFCtWjXV1klFpyjqBqAPl5kzZyI2NhbVq1fH3bt3Hzth4Fl069YN9vb2iIiIQKtWrZCZmfnYljwA7Nq1C7dv38a6devQvn174+PJyclPfD9nZ2ds374dqampBbZennf/FrTO33//Pd/jhi68Z1nni1bix1wK0r17dwDI9+W7YMECAECPHj2euI6goCD88MMPWLRoEapUqQIfH588z/ft2xf79+/H9u3b8702PT0dubm5AGC8YC09Pf1pN+ORZTp16hQmT54Mc3PzJ1bWatWqoUOHDvj6669x7dq1fM8/aorxk3h7e6NChQr497//Da1Wq8o6qegURd0A9F02Hh4eiIqKQlRUFGrUqJHny1wNZcqUQWBgIKKjo7Fy5Up4eHigUaNGj32NoaXwYMsgJycHX3755RPfr3fv3hAR40XRDzKsT439+7Du3bvj4MGD2L9/v/Gxe/fuYfny5XB0dET9+vWfep0vWqltuTRu3BiDBw/G8uXLjc3ggwcPYtWqVfDz80PHjh2fuI633noL77//PtavX493330XFhYWeZ6fPHky4uLi0LNnTwQHB6N58+a4d+8ejh8/jrVr1+LChQuoWrUqypUrh/r16yMqKgqurq6ws7NDw4YNn6mPtEePHqhSpQpiYmLg4+ODatWqPfE1X3zxBdq2bQsPDw8MHz4cTk5OuHHjBvbv348rV67g6NGjT1WGChUqYNmyZQgKCkKzZs3Qv39/2Nvb49KlS9iyZQvatGmDpUuXPvW2UdEoirph0K9fP0yfPh1WVlYYOnToY7urnpXhMoDExETMmzfvict7enqicuXKGDx4MMaOHQuNRoPw8PBCdUN17NgRQUFBWLJkCZKSktCtWzcoioKffvoJHTt2xOjRo1XZvw+bMmUK1qxZAx8fH4wdOxZ2dnZYtWoVkpOTERsb+0L263Mz3US152OYXvjLL78UuIxWq5WZM2dK3bp1xcLCQmrXri1Tp07NM9VQJP90ywd1795dAMi+ffse+XxGRoZMnTpVXFxcpGzZslK1alXx9PSUzz77THJycozL7du3T5o3by5ly5bNMy158ODBYm1tnW+9oaGhUtDHM2rUKAEgkZGR+Z4raCri+fPnZdCgQfLKK6+IhYWF1KpVS3r27Clr1641LlPQPk1MTBQAkpiYmO9xb29vqVixolhZWYmzs7MEBwfLoUOHHlluKhrFpW6IiCQlJQkAASB79uwpsKwPT0Xu0aNHvmUfV5YGDRqImZmZXLlypVDvsXfvXmndurWUK1dOatasKe+//75xmvWDx/nDU5FFRHJzc+XTTz8Vd3d3KVu2rNjb24uPj4/8+uuvxmUKu3+fZlvPnz8vffr0kUqVKomVlZW0bNlSNm/enGeZ4jQVWSNSwkeNXjB/f38cP34c586dM3VRjMaPH48VK1bg+vXrxeYeUfTyKU51o2nTprCzs8N///tfUxeF/lYM21LFx7Vr17Bly5YnDhAWpaysLKxevRq9e/dmsJDJFKe6cejQIRw5cgSDBg0ydVHoAaV2zOV5JCcnY+/evQgLC4OFhcUT79tVFFJSUrBz506sXbsWt2/ffuT9nYhetOJUN06cOIFff/0V8+fPR40aNQp1cSYVHbZcHuHHH39EUFAQkpOTsWrVqkdey1HUTp06ZZx+vGTJEjRp0sTURaKXUHGqG2vXrsWQIUOg1WqxZs0aWFlZmawslB/HXIiISHVsuRARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKorY+oCFHcigtu3b2P37t24dOkSfH194ejoCI1GY+qiEREVWxoREVMXorgREeh0OqSkpGDjxo2IjY3F0aNHodPp4OTkhIULF8LT0xPm5uamLioRUbHEbrEHGELl6NGjmDFjBjp37oxvvvkG7du3x65du7Bz507Y2dkhMDAQkZGR0Gq1YDbTy0ZEjP+o8F62/caWC/QfempqKg4cOICIiAgcO3YMderUwYABA+Dl5YVXXnkF5ubmEBHcunULq1atwjfffIOAgACMGzcO1atXN/UmEBUJRVGQlJSE2NhYuLq6wtvbGzY2NuwmfgwRQWZmJuLj45GUlAR/f3+4urrCzKx0n9u/tOEiIsjNzcWNGzcQHx+P6OhoXLx4EV5eXnjrrbfQsmVLWFtbP/K1Wq0WiYmJ+Oijj1CjRg3Mnj0bHh4epf5gKS4URcGNGzeg1WpRq1Ytdk8WARHB3bt3ERsbi6VLlyIlJQUajQYdO3bExIkT8eqrr6Js2bKmLmaxo9VqcfLkSSxcuBCJiYlQFAX29vYYO3Ys/P39UbFixdIbzPISURRFFEWRnJwcOXTokHzwwQfi5uYmzZs3l7lz50pSUpJotVpRFKVQ60pOTpYBAwaIu7u7REdHS3Z2dqFeS8/G8PllZWXJ22+/LT179pT79++bulilmqIokp2dLQkJCdKtWzdxc3OTTz/9VFJSUuTAgQPy1ltviYODg4SEhMilS5eMn9HLzLAPrl69KjNmzBAHBwfp27ev7N+/X27evCmLFi0Sd3d36dy5s+zcubPUfm+8NC0X+XvW1759+xAZGYnDhw/D3d0d/fr1Q5cuXVClSpVnOgNOT0/HihUrsHz5cgQEBGDChAmoWrVq6T0bMRERwZUrV6DT6VCrVi2Ehobixo0bCAsLQ2pqKu7du4c6deqYupiliqIoOHnyJL744gskJibi9ddfx+jRo+Hs7IwyZcpARPDXX38hISEBixcvxp07dzBq1Cj4+/vDzs7O1MU3mbS0NGzatAmff/45bGxs8K9//Quvv/46bG1todFokJubi+TkZHz55ZfYtm0bPD09MWbMGDRq1KhU9X6U6nAREWi1Wly9ehVbtmzB2rVrcfPmTXh5eSEwMBBNmzaFjY3Nc79Pbm4udu7cidDQUFSvXh1z5sxBgwYN2F2jIp1Ohzlz5mDXrl3w8/NDamoq7t69i+bNmyMmJgaOjo6YP38+ypTh7PrnpSgKrl+/ju+++w7ff/89HBwcMG7cOHh6esLS0jLf8iKClJQUrFmzBmFhYXBwcMD48ePh6emJcuXKvRQnWiKCrKws/Pzzz1i4cCHOnz+P4OBgBAUFoXr16o/cBzk5OThw4AAWLFiA8+fPo1+/fggODkaNGjVKRciUynAREeTk5OD48eNYs2YNtmzZAhsbG/Tv3x++vr5wdnaGRqNR9aAXESQnJyMkJAS//fYbZs+eDV9fX1hYWLwUletFUxQFiYmJOHDgAM6cOYPjx49DURQ0a9YM9erVQ5MmTdC5c+dHfvlR4Ri+IOPi4rBkyRLcu3cPkyZNgp+fH6ytrZ94HCuKggsXLmDJkiXYsGEDunTpgrFjx5b6Ey2dToezZ89i8eLF2LZtG3r27ImxY8fCxcXliSEhIrh//z42bdqEzz77DGZmZhgzZgwCAgJKfjAXfU/ci6Moity+fVtiY2MlICBAnJycJCAgQKKjo+XWrVuSm5v7wt8/LS1N/vOf/4irq6tMnTpVbt269ULf82Vi6P+Pi4uTnj17SufOnWXVqlWSk5NTKvusi1JOTo7s3r1bfH19pUGDBjJr1iz5888/RafTPdV6DGNie/bskT59+ki9evVkzpw5cvny5VL3GRnGVebOnSuurq7i7+8vP/74o9y/f/+pt1Wn08n169flk08+kQYNGoiPj4/s2rVLsrOzX1DpX7xSES45OTly7tw5Wbp0qbRp00YaN24s48ePl/3790tGRoZJyrN582Zp1aqV9OrVS44ePfrCg+1loNPpJCEhQTw9PSU8PFy+++47adGihWzbto379xnl5ubK77//LuPGjZP69evLkCFD5PDhw6LVap973Xfu3JGoqChp06aNvPbaaxIeHi6pqamlImTS0tLk+++/N27b6tWrJS0t7bm3TavVyrFjx2TEiBHSoEEDGTVqlJw+fbpEHt8lNlwURZHMzEw5ePCgTJ06Vdq1ayf9+vWTpUuXyvnz50Wn05nsIDbMFklKSpJ+/fqJh4eHREdH8wz7OWVlZckHH3wgfn5+kpaWJikpKRIQECCTJ0+We/fumbp4JYpOp5Nbt27J4sWLpUGDBtKlSxeJj49X9Rg11IPr16/LvHnzxMXFRXx9fY1n5CWtLhhazrt375bevXuLs7OzfPzxx/Lnn3+qOktOURTRarWyY8cO8fHxEXd3d/nss88kJSXlqVuSplSiwsXwAaampsqGDRskKChIWrZsKSNHjpT4+Hi5c+dOsdr5iqLI3bt35ZNPPhEXFxf56KOPSs2ZmykoiiLp6emSnJxsPHk4d+6cZGRkcJ8WkqHbav369dKmTRtp0qSJrFixQpWz7sfRarVy5swZGTlypDg5Ock3kyaJ7vRpkZLyuSmKaJOSZNmUKVK3bl0ZPny4nDhxQpUWXsFvqT/eV65cKc2bN5fXXntNoqOjn6nbzRRKzIC+VqvFhQsX8MMPPyA+Ph5arRaenp7w8/NDvXr1ivXgV05ODnbs2IGZM2eiRo0amDVrFi+6fEqKoiAlJQVlypRBlSpViu1nXZxptVqcOHECX331Ffbv3w8/Pz8MGTIEDg4ORXIsyt8TBn7dvx+2X32FRklJ0AwaBAQGAtWrA8XxMxUBUlKAiAgo4eHYX68eckeOxD9bty6y7xxFUXD58mWEhYUhLi4Onp6eGD58OBo2bFisL1xVPVzk7/tzAYC5uflz7Xz5+7YJJ0+exNq1a3Hw4EHY2tqiT58+6NSpE2rVqlVivqBFBOfOnUNISAiSkpIwbdo0dO/enbObHkP+nkp++fJlbNy4Ebt27UL79u0xceJEhstT0Ol0uHbtGlatWoXt27ejVatW6N+/Pxo3bmy6qdt37wLr1gGffw7Y2ABjxgDe3vr/Lw6frQhw7x7w3/8CixYBaWnAv/4FvPkmUKmSSYqk0+lw/PhxRERE4Oeff0a3bt0wePBg1KhRo1jOxlM9XO7fv49vv/0WFStWRL9+/Z764DUUJy0tDQkJCYiLi8Pvv/+Oli1bwtfXF6+99prxjKGkfcGICDIyMrB06VKEhYVh4MCBmDBhQum+BcRTMnz+OTk5OHjwIDZv3oyEhAQ4OjrijTfegI+PD1suhSR/X+QYFxeHsLAw2NnZYfjw4ejUqZPpp8iL6P9dvw4sXw6sXg00awaMHw+0bAmYmZkmZEQARQF+/VUfKgcOAP37A+++C9SsqS+TCfeb4YRr9+7d+Oabb3Dt2jUMHToUfn5+qFChQrGqF6qFi06ng5mZGXQ6HXx9feHu7o5PPvnEGC6F2WitVovz589j586d2Lx5MzQaDdq2bQt/f3/UrVsXVlZWxWrnPQvDwbFjxw7MmDEDderUwfTp09GoUaMSv23Py3Dx3v79+xETE4MbN26gYcOG6NOnDxo1aoRKlSq99PuosO7fv4/9+/fj66+/RlpaGt566y307NmzeAazVgucOqX/Mv/5Z6BnT2DYMKBePX3IFBUR4Px54P/+T9+q+uc/gXHjgEaNAAuLoitHIcjfN9vdunUrIiIiYGtri+HDh6Nt27YoX768qYsHQKVwERGsXLkStra26NSpEyZPnoxGjRohMDAQP/74I6pWrYoOHTo88qA2nF0dO3YMsbGxOHz4MOzs7NC7d2+0b98eNWvWLDFdX0/DcHfZ0NBQnD17FjNmzIC3t/dL100mIsjOzs7T9ZWZmQlvb290794dbm5uxbpfuTiRv2/GmpSUhLCwMBw4cABdu3bFwIEDjRcOF2v37wMJCcCCBcCdO0BwsH48xs7uxbYWRPTdXmvXAitWAJaWwKRJQJcuQDH5oi6IiODChQuIjIzEli1b0KJFC7zzzjtwc3NDmTJlTPqZqxIu9+/fx5gxY7Br1y54eHggOzsblStXxs2bN3H58mUEBgbio48+MvYLyt+/aZCammpspZw5cwZt27ZFr169jF1fQOFaPCWV/H2n2aVLl2L58uV4++23MXbs2FJ/hm445LKysvDLL79g/fr12LNnD5ycnODr64uuXbsaz7BL835Qk4jg5s2bWL16NWJiYtCwYUOMGDECTZo0ee6xzyIloh+PiYwEli0DqlQBJkwAunbVf+mruR0iQE4OkJgIfPqpvovu3XeBgQOBihWLx9hPIRjGuY8fP46vvvoKR44cQe/evTF48GBUq1bNZJ+9KuGiKAru3LmDlJQUJCYmYtmyZahYsSImTZqExo0bo3Llysabtmm1WiQlJWHbtm3YsmULLC0t0aFDB/j5+aFOnTqwtLQsORVBJdnZ2di5cydmzJiB2rVr4+OPP4abm1upbbFdv34du3fvRmxsLG7evIkmTZqgd+/eaNy4sfE4edmJCG7cuAGdToeaNWsWuE8M43jbtm3DihUrYG1tjSFDhqBTp04oX758ydyXhnGPy5eBL78ENm8GmjcHxo4FmjYFHjWOawiKlBR9C6hcOaBaNaBs2UeHhE4HHDumn1Cwfz/g4wOMHg04OJhuvOc5GSZA/fTTT1ixYgXS0tIwdOhQ+Pj4PHZcV0Rw9epVWFhYqBpGqo25GM6cQkJCYGdnhytXrsDd3R3jxo2DjY0NRARnz57F0qVLcerUKdjb26NPnz5o3bp1qe36ehoiglOnTmHmzJk4d+4cpk+fDh8fH5QtW7ZkfkE8wND1lZycjE2bNmHXrl3Izc3F66+/jh49esDFxQUWxaxP25QURYGIYNmyZbhz5w4+/PBD43MPHgv379/HkSNH8PXXX+Py5cvo3bs3AgICCrxRYomk1QKHDunHY06c0M/WCg4G6tT533hMRgawaZN+UsClS/qQKVtWHxQDB+rHcGxt9csqCnDlCrByJRAdDbi768dVWrUqduMqz8rwXbxx40ZERUWhRo0aGD58OP75z38ae4QMy4kINBqN8TKJYcOGwczMTJXjR7VwMVSCo0ePYtWqVTh79izee+89vPfeexg5ciTMzMxw+PBhxMXFoWPHjmjVqtVL2Up5HMNZ6JIlSxAWFoYhQ4Zg3LhxxW4WSGEYDtzMzEz88ssv2LBhAw4cOABnZ2f4+vqic+fOxtuyl7Rte5EURcGaNWvg4uKCI0eOIDMzE6+//jrOnDmD7t27w8rKyniDyLCwMOzYsQNdu3bFsGHD4OTkBKAU7k8RIDsb2LpV332l1epbHK1b67vQQkOBr74CGjQAunfXt1iuXwe2bAF+/x0YNUq/jK2tvpUyfjxgbq7/b8+egJVViWypPI7haz05ORkrV67E1q1b0bFjRwwfPhzOzs4wMzPD/fv3sW7dOjRo0ABbt27FP/7xDzRq1AgXLlyAr6/vc5/wqxYu2dnZiI6ORm5uLgYNGoT79+9jwYIF6NSpE9q0aQONRgOdTgdFUUw+0FTc5eTkYPv27ZgxYwYcHR0xZ84cuLu7l5h9JiL4888/kZCQgHXr1uH27dto2bIlAgIC0Lhx45LbXVMEMjMzsWjRIpw8eRJ//fUXAKBy5cpwdnbG2LFjkZOTg5iYGERGRqJevXoYMWIEmjZtWipauE+kKMDt2/qQadMGcHYGvvlG3/Lw99cHT/Xq+haNTqcPmHHj9Mt//jnw9ttAUhKwZ48+VOztS12oPEz+vkP8iRMnsGzZMpw5cwZ9+/ZF3759YWVlhQULFuCPP/5Aeno6LC0tYWNjgwYNGmDMmDF5WjnPQtXrXBRFgVarhaWlpXGjXoqD/gUQEZw8eRIzZ87EH3/8gdDQ0GI9m8xw9fW5c+cQFxeH3bt3w9zcHN7e3vDx8YGjoyNnfRVSTk4ODh8+jEWLFiEzMxOTJ09Gy5YtUaZMGSxduhQ7duxAcHAwOnfu/HJeI2X4ysrIAF5/Xd/N9eOP+rB52JkzQMeOgJMTsG2b/iJNoNSHysMMvSI7duzAt99+a/zhN61Wiz179mDp0qWwtLTEBx98gIYNG6rSTV1ibv/yMhIR3LlzBwsXLkRkZCQGDRqE9957r1gNeiuKgrt372Lfvn3YsWMHDhw4ABcXF/j7+6N9+/bs+noG165dw8SJE+Ht7Q2tVott27bhiy++QLVq1Yy/XW9vb899+ssvQK9egJcXEB6uH2d5WHa2fjrzzz/rx2WaNy/6chYjhvEYEUG1atVw8eJFjBs3DsHBwTh//jyOHz+OBQsWqPJLovzZvmJMo9GgYsWKCAkJQbNmzTB9+nQcO3YMc+bMgaurq3GZomYYT7l48SI2b96MqKgoZGRkoG/fvli0aBE8PDxKxQWvplK+fHl07doVfn5+yMrKQtmyZY1didWrVzd18YqP69f1t2hxddWPoTyKuTng5gbs3AncuFG05SuGNBoNqlWrZvzb1tYWvXv3RocOHdC6dWvEx8er1jvClksJodPpcPr0aYSGhuLSpUsICQlB9+7di3yWVWZmJk6cOIGYmBjs2LED1tbW8PPzg5+fHxwcHEx/W5FSQlEU4340zOjhfn3I2rVAUBAwbRowZcqjr+bX6YB//xuYOxeIiNCPzZDRgzPGDH+rNXOXLZcSwtzcHA0bNkRYWBgWLlyI999/H6dOncLIkSNf+EWXiqIgPT0de/fuRUREBH777Te4ubnho48+QocOHVC1alV+8answQrOfVuAcuX004czMv43DvMww0WZFhbF/mp7U3j4pEXNY43hUsJUqlQJ06ZNQ9OmTRESEoJff/0Vc+fOhYuLi6oHxoNdXxs2bEBUVBTS0tLg4+OD8PBwNG7cmFPJybRq1dJPLz51St9CeVTXmE6nf75CBf2NJ6nIsFushFIUBadPn0ZISAiuXLmC0NBQdOvWTZVbqN+7dw9HjhxBdHQ0du7ciapVq8LX1xe+vr6oXbs2u76oeMjM1F/Xcvq0/p5kDRrkX+bIEf09wpo00Q/oP+f0Wio8hksJJiK4ffs2Pv/8c0RGRmLYsGEYPnz4M8300Ol0uH37Nnbv3o3IyEicPn0ar776KgYNGgRPT0/OTqLiRwT4/ntgxAj9dOP58/VX5ZcpA+TmAsnJ+utc9u7V35Cyd++XbgqyKTFcSoGcnBxs2bIFISEh8PDwwJw5cwp1F1zDDe8uXbqE2NhYrF27FqmpqfDz80OfPn3QrFmzp/rJBKIid++e/uLJTz/V3xKmc2f9Ffo3buh/6OvqVWDqVH3IcMylSDFcSglFUXDmzBmEhITg4sWLmD17Nrp27VrgbLLMzEwcOnQI0dHR+OGHH1C7dm306tULAQEBqFGjBu+iQCWH4Vb9ERHAuXNAVpa++6tePf29xTp00N/ihYoUw6UUMVwgtWTJEsTExGDw4MEYNmyYsUtLp9MhJSUFP/30E8LDw3HhwgXUr18fQUFBaNWqFWd9UclluCtyaqr+wklLS/3t+i0s2BVmIgyXUsbwg1Hr1q3DxIkT4e7ujnnz5sHKygobNmxAXFwc0tPTERAQgICAADRt2rRk/d4HEZUIDJdSSqfTYd++fZg4cSKSkpJgZmZm/HXQXr16oWrVquz6IqIXhuFSij14nYqDgwPatGnDWV9EVCQYLkREpLqX++cfiYjohWC4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKQ6hgsREamO4UJERKpjuBARkeoYLkREpDqGCxERqY7hQkREqmO4EBGR6hguRESkOoYLERGpjuFCRESqY7gQEZHqGC5ERKS6/wd96EgQ+8p2ywAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 500x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "smiles_list = [\"[*]CC[*]\", \"[*]COC[*]\"]\n",
        "mol_list = [Chem.MolFromSmiles(smiles) for smiles in smiles_list]\n",
        "\n",
        "def show_mols(mols: list, labels: list, figsize=(5,5)):\n",
        "    \"\"\"\n",
        "    This function is used to display a list of molecules using matplotlib.\n",
        "    \"\"\"\n",
        "    if len(mols) > 1:\n",
        "        fig, ax = plt.subplots(1, len(mols), figsize=figsize)\n",
        "        for i in range(len(mols)):\n",
        "            mol = mols[i]\n",
        "            label = labels[i]\n",
        "            img = Draw.MolToImage(mol)\n",
        "            ax[i].imshow(img)\n",
        "            ax[i].set_title(label)\n",
        "            ax[i].axis('off')\n",
        "            ax[i].set_aspect('equal')\n",
        "    else:\n",
        "        mol = mols[0]\n",
        "        label = labels[0]\n",
        "        img = Draw.MolToImage(mol)\n",
        "        plt.imshow(img)\n",
        "        plt.title(label)\n",
        "        plt.axis('off')\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "show_mols(mol_list, [\"Polyethylene\", \"Polyvinylalcohol\"])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTcQDGERTnbO"
      },
      "source": [
        "#### Generating Fingerprints"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Initially, we will load tokenizer and model weights for PolyBERT using HuggingFace transformer library."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "d:\\own\\career\\open_source\\env_files\\lib\\site-packages\\transformers\\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "# Step - 0: Load the tokenizer and model weights for PolyBERT\n",
        "\n",
        "# Load model from HuggingFace Hub\n",
        "tokenizer = AutoTokenizer.from_pretrained('kuelumbus/polyBERT')\n",
        "polyBERT = AutoModel.from_pretrained('kuelumbus/polyBERT')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Once initialized, we can define our PSMILES string and make the finger print predictions as follows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "model output >> BaseModelOutput(last_hidden_state=tensor([[[-0.3600, -0.1442, -0.9196,  ..., -0.5034,  0.5028,  1.2998],\n",
            "         [-0.3190,  0.0676, -0.9040,  ..., -0.3021,  0.5724,  1.2594],\n",
            "         [-0.0568, -0.4151,  0.0570,  ..., -0.3222,  0.4901,  1.7791],\n",
            "         ...,\n",
            "         [ 0.0989,  0.1656, -0.0327,  ..., -0.3214, -0.0025,  0.6070],\n",
            "         [-0.2276,  0.3628, -0.5948,  ..., -0.8587,  0.1078,  0.0966],\n",
            "         [ 0.2695,  0.6533,  0.5087,  ..., -0.1991, -0.1496,  0.5560]],\n",
            "\n",
            "        [[-0.4281,  0.6151, -0.6989,  ..., -0.4487,  0.4334,  1.0624],\n",
            "         [-0.4327,  0.7250, -0.7236,  ..., -0.4047,  0.4732,  1.0544],\n",
            "         [-0.3124,  0.1882,  0.1288,  ..., -0.6645,  0.2649,  1.5900],\n",
            "         ...,\n",
            "         [ 0.0755,  0.8277, -0.1443,  ..., -0.1048,  0.1284,  0.3646],\n",
            "         [-0.3230,  1.1286,  0.3890,  ..., -0.7890,  0.2660,  1.9148],\n",
            "         [-0.3051,  0.0756,  0.1737,  ..., -0.8654, -0.0336,  0.7049]]]), hidden_states=None, attentions=None)\n",
            "output shape >> torch.Size([2, 8, 600])\n"
          ]
        }
      ],
      "source": [
        "# Step - 2: Tokenize the PSMILES and generate the fingerprint\n",
        "# Sentences we want sentence embeddings for\n",
        "psmiles_strings = [\"[*]CC[*]\", \"[*]COC[*]\"]\n",
        "\n",
        "# Tokenize sentences\n",
        "encoded_input = tokenizer(psmiles_strings, padding=True, truncation=True, return_tensors='pt')\n",
        "\n",
        "# Compute token embeddings\n",
        "with torch.no_grad():\n",
        "    model_output = polyBERT(**encoded_input)\n",
        "    print(\"model output >>\", model_output)\n",
        "    print(\"output shape >>\", model_output.last_hidden_state.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The model output is 3 dimensional, while the fingerprint has to be of shape `1 X 600`. The first array in the output contains all the token embedding. To normalize the output, we will clip the values same as the Attention Mask. Then we will form a finalized vector by find mean of 8 token embedding layers ( as the shape is `2 X 8 X 600`). This process is called Mean Pooling. The same has been implemented as follows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KvnpL-jjbdGD",
        "outputId": "2d1afa72-9bf9-486f-941a-0f87a8af50eb"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "d:\\own\\career\\open_source\\env_files\\lib\\site-packages\\huggingface_hub\\file_download.py:159: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\USERNAME\\.cache\\huggingface\\hub\\models--kuelumbus--polyBERT. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
            "To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
            "  warnings.warn(message)\n",
            "d:\\own\\career\\open_source\\env_files\\lib\\site-packages\\transformers\\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
            "  warnings.warn(\n",
            "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fingerprints:\n",
            "tensor([[-0.1186,  0.1891, -0.5370,  ..., -0.8715,  0.2979,  0.6712],\n",
            "        [-0.3981,  0.4833, -0.2797,  ..., -0.5428,  0.2027,  0.9449]])\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Step - 3: Performing the mean-pooling (to normalize the fingerprint value)\n",
        "\n",
        "#Mean Pooling - Take attention mask into account for correct averaging\n",
        "def mean_pooling(model_output, attention_mask):\n",
        "    token_embeddings = model_output[0] #First element of model_output contains all token embeddings\n",
        "    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()\n",
        "    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)\n",
        "\n",
        "# Perform pooling. In this case, mean pooling.\n",
        "fingerprints = mean_pooling(model_output, encoded_input['attention_mask'])\n",
        "\n",
        "print(\"Fingerprints:\")\n",
        "print(fingerprints)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "73Ju_yGCWnID"
      },
      "source": [
        "We can notice two-dimensional tensors that hold the fingerprint for each polymer. Let's vailidate the dimension if those have the required dimension as specified."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8wtCV_yzXBFf",
        "outputId": "0c92fe26-a9e3-40eb-cb73-fcf0ffd4f51f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "torch.Size([600])\n",
            "torch.Size([600])\n"
          ]
        }
      ],
      "source": [
        "for f in fingerprints:\n",
        "    print(f.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LEGwO9QQXo7r"
      },
      "source": [
        "As per the above output, we can see dimension is `1 X 600` same as mentioned in section 3. Now we can use these vector to predict similarity between 2 polymers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3pDQu7QEk3LR"
      },
      "source": [
        "### 5. Similarity Predictions Using Fingerprint <a id=\"similarpred\"></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7nhokBwY9IM"
      },
      "source": [
        "There are various algorithms to find similarity between different high dimensional vectors. We will be using **cosine similarity** for our use case.Cosine similarity is a measure of how similar two vectors are, regardless of their size. Imagine each vector as an arrow pointing in space. Cosine similarity compares the angle between these arrows, not their length. If the angle is small (close to 0°), the vectors are similar; if the angle is large (close to 90°), they are less similar. We can visualize the same using following image.\n",
        "\n",
        "![similarity](https://storage.googleapis.com/lds-media/images/cosine-similarity-vectors.original.jpg)\n",
        "\n",
        "\n",
        "In simple terms, cosine similarity tells us how much two items \"point\" in the same direction, making it useful for comparing things like text documents or molecular fingerprints where exact values matter less than overall patterns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vN0ndpq0ijgm",
        "outputId": "2798d1bf-e4c9-48f7-cb53-8f9c3f6a5293"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Cosine Similarity between fingerprints: '[*]CC[*]' and '[*]COC[*]' is \n",
            "0.8462606072425842\n"
          ]
        }
      ],
      "source": [
        "import torch.nn.functional as F\n",
        "# Compute cosine similarity between the first and second fingerprint\n",
        "cosine_sim = F.cosine_similarity(fingerprints[0].unsqueeze(0), fingerprints[1].unsqueeze(0))\n",
        "\n",
        "print(f\"\\nCosine Similarity between fingerprints: '{psmiles_strings[0]}' and '{psmiles_strings[1]}' is \")\n",
        "print(cosine_sim.item())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcJR1znBcmgN"
      },
      "source": [
        "By determining the cosine similarity of the fingerprint vectors, we found that polyethylene `([*]CC[*])` is similar to polyvinylalcohol `([*]COC[*])` by 85%."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MvqTzEjOk7ku"
      },
      "source": [
        "### 6. Summary <a id=\"sum\"></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YwDAUZP1gpP6"
      },
      "source": [
        "The tutorial provides an in-depth exploration of PolyBERT, a state-of-the-art machine learning model designed to advance polymer informatics by leveraging natural language processing techniques. PolyBERT enables efficient and precise exploration of the vast chemical space of polymers, facilitating the identification of materials with specific properties. The model uses transformers latent space to convert polymer chemical structures into machine-readable vectors, known as fingerprints, which are crucial for various computational tasks.\n",
        "\n",
        "Key sections of the article include:\n",
        "\n",
        "1. **Introduction:** The introduction outlines the challenges in polymer discovery and introduces PolyBERT as a novel solution, highlighting its applications in cheminformatics and multi-task property prediction.\n",
        "\n",
        "2. **Overview of PSMILES and Fingerprints:** This section discusses Polymer SMILES (PSMILES), a specialized notation for representing polymer structures, and chemical fingerprints, which are essential for tasks like similarity searches and machine learning.\n",
        "\n",
        "3. **Understanding PolyBERT Latent Space:** PolyBERT's latent space, based on the DeBERTa architecture, is explained as a lower-dimensional representation that captures the essential features of polymer structures. This space is crucial for generating accurate fingerprints and performing prediction tasks.\n",
        "\n",
        "4. **Implementing PolyBERT Using HuggingFace:** The implementation section guides readers through using the HuggingFace library to generate fingerprints for polymers, using specific PSMILES strings as examples.\n",
        "\n",
        "5. **Similarity Predictions Using Fingerprint:** The article demonstrates how to compute cosine similarity between fingerprint vectors, illustrating how PolyBERT can assess the similarity between different polymers.\n",
        "\n",
        "By combining the advanced capabilities of transformer models with the unique challenges of polymer informatics, PolyBERT represents a significant leap forward in the field, offering scalable solutions for material discovery and prediction."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qqla9MHjk-av"
      },
      "source": [
        "### 7. Reference <a id=\"ref\"></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1n6qbQKZ7KUU"
      },
      "source": [
        "1. [Kuenneth, Christopher, and Rampi Ramprasad. \"polyBERT: a chemical language model to enable fully machine-driven ultrafast polymer informatics.\" Nature Communications 14.1 (2023): 4099.](https://www.nature.com/articles/s41467-023-39868-6)\n",
        "2. [He, Pengcheng, et al. \"Deberta: Decoding-enhanced bert with disentangled attention.\" arXiv preprint arXiv:2006.03654 (2020).](https://arxiv.org/abs/2006.03654)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Congratulations! Time to join the Community!\n",
        "\n",
        "Congratulations on completing this tutorial notebook! If you enjoyed working through the tutorial, and want to continue working with DeepChem, we encourage you to finish the rest of the tutorials in this series. You can also help the DeepChem community in the following ways:\n",
        "\n",
        "## Star DeepChem on [GitHub](https://github.com/deepchem/deepchem)\n",
        "This helps build awareness of the DeepChem project and the tools for open source drug discovery that we're trying to build.\n",
        "\n",
        "## Join the DeepChem Gitter\n",
        "The DeepChem [Gitter](https://gitter.im/deepchem/Lobby) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!\n",
        "\n",
        "## Join the DeepChem Discord\n",
        "The DeepChem [Discord](https://discord.gg/cGzwCdrUqS) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Citing This Tutorial\n",
        "If you found this tutorial useful please consider citing it using the provided BibTeX."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "@manual{Intro1,\n",
        " title={PolyBERT: Introduction to Chemical Language Models for Fingerprint Generation},\n",
        " organization={DeepChem},\n",
        " author={Mohanty, Debasish},\n",
        " howpublished = {\\url{https://github.com/deepchem/deepchem/blob/master/examples/tutorials/PolyBERT___Introduction_to_Chemical_Language_Models_for_Fingerprint_Generation.ipynb}},\n",
        " year={2024},\n",
        "}"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
